node:quic: report a datagram's status under the id kept in its lsquic frame - #44680

Other changes a user can see

• stats.datagramsSent counts a datagram when lsquic frames it. On main the count came one event-loop pass later.
• Each id that datagramsSent counts ends in datagramsAcknowledged or datagramsLost on a session that continues. On main a datagram that lsquic dropped before the send was counted in neither.
• An ondatagramstatus that destroys the session (a throw does) ends the statuses of that pass. On main each further datagram of the same packet raised an uncaught ERR_INTERNAL_ASSERTION ("Unexpected datagram status event").
• A datagram of a rejected 0-RTT flight gets no status, as in Node. The ids after it now keep their own statuses.
• The diagnostics channel quic.session.receive.datagram.status gets the same corrected pairs.

The lsquic change

patches/lsquic/node-quic-accessors.patch is a patch file, so its diff here is a diff of a diff. This is what it changes in lsquic, condensed:

/* include/lsquic.h */
-    void (*on_datagram_status)(lsquic_conn_t *c, unsigned count, int acked);
+    void (*on_datagram_status)(lsquic_conn_t *c, uint64_t dgram_id, int status);
+void lsquic_conn_set_datagram_id (lsquic_conn_t *, uint64_t dgram_id);  /* only inside on_dg_write */
+enum lsquic_dg_status { LSQ_DG_ACKED, LSQ_DG_LOST, LSQ_DG_UNSENT };

/* lsquic_full_conn_ietf.c: struct ietf_full_conn */
+    uint64_t                    ifc_dg_id;
-    int                         ifc_ping_us_override_set;  /* also ifc_datagram_early, ifc_abort_silent */
+    unsigned char               ifc_ping_us_override_set;

/* lsquic_full_conn_ietf.c: write_datagram() */
+    conn->ifc_dg_id = 0;
     w = pf_gen_datagram_frame(..., on_dg_write, &conn->ifc_conn);
-    lsquic_packet_out_add_frame(packet_out, mm, 0, QUIC_FRAME_DATAGRAM, ...);
+    lsquic_packet_out_add_frame(packet_out, mm, conn->ifc_dg_id, QUIC_FRAME_DATAGRAM, ...);

/* lsquic_send_ctl.c */
+static void
+send_ctl_report_datagrams (struct lsquic_send_ctl *ctl,
+        struct lsquic_packet_out *packet_out, enum lsquic_dg_status status)
+{
+    struct packet_out_frec_iter pofi;
+    struct frame_rec *frec;
+
+    if (!(packet_out->po_frame_types & QUIC_FTBIT_DATAGRAM)
+            || !ctl->sc_enpub->enp_stream_if->on_datagram_status)
+        return;
+
+    for (frec = lsquic_pofi_first(&pofi, packet_out); frec;
+                                                frec = lsquic_pofi_next(&pofi))
+        if (QUIC_FRAME_DATAGRAM == frec->fe_frame_type)
+            ctl->sc_enpub->enp_stream_if->on_datagram_status(
+                ctl->sc_conn_pub->lconn, frec->fe_u.data, status);
+}
 send_ctl_handle_regular_lost_packet():  LSQ_DG_LOST    (replaces the count report)
 lsquic_send_ctl_got_ack():              LSQ_DG_ACKED   (replaces the count report)
 lsquic_send_ctl_next_packet_to_send():  LSQ_DG_UNSENT  (new, in the PO_REPACKNO block)
 lsquic_send_ctl_squeeze_sched():        LSQ_DG_UNSENT  (new, in the drop arm)
-lsquic_packet_out_count_datagram_frames()                (removed)

• The two LSQ_DG_UNSENT sites are the places where lsquic discards a DATAGRAM frame that it framed and did not send. After a retransmission timeout lsquic sends at most two packets until an ACK arrives. It drops a waiting packet that holds only datagrams (lsquic_send_ctl_squeeze_sched). It cuts the DATAGRAM frame out of a waiting packet that it gives a new number (PO_REPACKNO).
• No report is made for a rejected 0-RTT flight, for lsquic_send_ctl_drop_scheduled, or at connection cleanup. Node reports nothing there.
• The patch file was regenerated from the patched tree. The later patches (coalesce-batch-drop, connection-close-pns, goaway-accept-uni-streams) apply on top of it unchanged.

Designs that were weighed

• The frame record with only the two existing report sites (acknowledged, lost). It fixes the report in the title. It leaves 9 of 13 ids without a status in the outage probe below, and datagramsSent stays 13 against 3 acknowledged and 1 lost.
• A map from packet number to ids in session.rs (the idea in node:quic on lsquic — Node v26 compat, HTTP/3 #32602). It is a second book beside the packet, and a packet that lsquic drops or renumbers leaves it stale.
• One report site in send_ctl_destroy_packet. It adds a bit test for each destroyed packet on the ACK path of every connection, and it fires at close.
• Not to frame a datagram that lsquic cannot send (Node's behaviour). This is the better end state and a larger lsquic change. It is listed under follow-ups.

Measurements

Release builds of the merge base 13a98b0 and of this branch, linux x64. perf, valgrind, strace and bloaty are not installed on this machine, so the instruction counts are static counts from objdump.

• struct ietf_full_conn, release: 10232 -> 10232 bytes (the calloc immediate in lsquic_ietf_full_conn_client_new and _server_new). Debug: 10280 -> 10280. Bun.serve HTTP/3 and fetch use this struct too.
• Instructions, main -> PR: lsquic_send_ctl_sent_packet 134 -> 134, squeeze_sched 129 -> 224, next_packet_to_send 266 -> 362, got_ack 963 -> 1001, write_datagram 99 -> 99. Each added instruction is behind the QUIC_FTBIT_DATAGRAM test.
• Release text: 80,732,991 -> 80,732,767 bytes (-224, size). lsquic_send_ctl_got_ack 4123 -> 4367 bytes, write_datagram 405 -> 418 bytes.
• QuicSession 880 -> 848 bytes. SessionEvent 136 -> 136 bytes.
• Session events for 1000 datagrams sent 5 ms apart on a lossless path: 2000 -> 930. Id queue operations: 2000 -> 0. Burst of 2000 datagrams: 2000 sent events and 1013 status events -> 0 and 516.
• Entries into the two new report sites in a lossless Bun.serve HTTP/3 run of 2,247 sent packets: 0.
• Allocations for a datagram: none added (read from the code: the id uses a slot that exists). UDP packets for 1000 lossless datagrams: 1057 to 1061 -> 1055 to 1060. Outage probe: 4 -> 4 datagram packets on the wire.
• Outage probe (13 datagrams around a retransmission timeout): ids without a status 9 -> 0, ids that contradict the server 2 -> 0, datagramsSent/Acknowledged/Lost 13/3/1 -> 13/3/10, in 3 of 3 runs.
• Loss of datagram 1 of 6: 2 of 6 ids wrong -> 0.
• An app that resends each datagram that is reported 'lost', in a 3 s outage. With one chain of resends: 2 -> 11 to 13 'lost' callbacks. With its own datagram each 10 ms too: 85 or 86 -> 805 to 1040 (5 runs). The count depends on timing: an earlier build of this branch gave 466 to 500. The packets on the wire are the same (88 to 90 datagram packets).

Tests

test/js/node/quic/quic-datagram.test.ts puts a UDP relay between a client and a server in one process. The relay loses or keeps packets that it selects by length. This is the loss-injection harness that #32602 said the fix needs.

• The fifth test pins the new rule that joins consecutive ids into one event only when the status is the same. Without that rule it fails in 6 of 6 runs, and in 15 of 15 runs on a loaded machine.
• Each failure on main takes less than 0.4 s. It is an assertion diff, or in the ninth test the uncaught ERR_INTERNAL_ASSERTION. No test fails by a timeout.
• Runs of the whole file on this branch: release 100 of 100, and 60 of 60 with 24 busy loops on 12 cores. Debug (ASAN) 20 of 20, and 15 of 15 with the same load. The loaded runs use the per-test timeouts of CI (90 s, and 270 s on ASAN). With the default 5 s, the loaded debug build times out while it loads node:quic. The later commits change only the test file: how the tests fail, how they release what they open, and their types. After them: debug 12 of 12, and 10 of 10 with the load. The release runs are from before them.
• A session that fails in the middle of a test rejects the wait that the test is in. Checked with a copy of the file that destroys the client or the server session: the test fails in about 0.55 s with that error. A handshake that fails releases the endpoints, the relay sockets and the sessions (checked with an ALPN mismatch: the test fails in less than 0.5 s with the transport error).
• The file passes the type check of test/ (bun check js/node/quic/quic-datagram.test.ts in test/: no type errors).
• The three "before the send" tests need a retransmission timeout of lsquic, which is a wall-clock timer of 200 ms or more. Their assertions hold on each path that lsquic can take, and no wait in them depends on the path. Whether they reach the two new report sites depends on timing: lsquic sends a NEW_CONNECTION_ID frame each second in the first seconds of a connection, and a slow run can leave the client with other retransmissions that hold back new frames. A stricter copy of the cut-out test, which fails if the datagram is not cut, passes 30 of 30 on the release build (idle and loaded), 20 of 20 on the debug build, and 13 of 20 on the debug build under the load above. gdb breakpoints on an idle machine: 2 hits at the drop site in each "dropped before the send" test, 1 hit at the renumber site in the cut-out test.
• Other suites on this branch: the 235 vendored test/js/node/test/parallel/test-quic-*.mjs (same result on both release builds: 233 pass, test-quic-stream-idle-timeout.mjs and test-quic-h3-stream-idle-timeout.mjs fail on both), test/js/node/quic/ (30 pass, debug), serve-http3.test.ts (72 pass, 1 skip), fetch-http3-client.test.ts (65), fetch-http3-adversarial.test.ts (27), fetch-http3-cold-post.test.ts (1 pass, 1 skip).
• Not run: macOS and Windows. The change has no platform-specific code.

Cases that stay as they are

• lsquic arms no loss timer for a packet that holds only DATAGRAM frames. A lost datagram gets its status when a later packet of the session is acknowledged. The last datagram of a quiet session gets none. Same on main.
• The retransmission timer of lsquic declares packets lost. It does not probe. A datagram that arrived can be reported 'lost' when its ACK is late. Same on main.
• lsquic never reports an acknowledgement after a loss. Node can report both for one id.
• 'abandoned' changes no counter. Node counts it in datagramsLost.

Follow-ups (not in this PR)

• lsquic: keep a framed DATAGRAM packet that cannot leave yet, and do not drop it. Then LSQ_DG_UNSENT and the resend cycle above go away, and bun matches Node.
• lsquic: a loss timer for packets that hold only DATAGRAM frames.
• One dead peer stalls the datagram sends of every session on an endpoint. udp: stop setting IP_RECVERR on UDP sockets #35922 touches this path.
• The session-ticket loop and the new-session loop have no destroyed check of their own. A stray callback from the ticket loop did not reproduce: the second ticket of a connection is delivered 500 ms after the first, in a pass that has the check. These are the two self-review concerns that this PR does not take.
• maxDatagramSendAttempts is validated in quic.ts and read nowhere.

Open PRs that touch the same lines

• node:quic: remove unsafe from the endpoint, session, stream and TLS code #40220 rewrites the same FFI thunks. The one that lands second applies the new signature again.
• Bun.serve: serve WebTransport datagrams over HTTP/3 #40027 adds a second on_dg_write user. It pays nothing if it does not set an id.